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USE  OF  THE  OCCUPATIONAL  KNOWLEDGE  TEST  TO  ASSIGN  EXTRA 
CREDIT  IN  SELECTION  OF  AIR  TRAFFIC  CONTROLLERS 


Current  practice  in  the  selection  of  air  traffic  control  (ATC)  trainees 
allows  assignment  of  extra  credit  for  certain  types  of  prior  experience  for 
applicants  who  successfully  pass  the  present  Civil  Service  Commission  (CSC) 

ATC  test.  The  extra  credit  gives  a candidate  a higher  CSC  rating  and  thus 
increases  the  prospect  of  being  selected.  The  idea  of  allowing  extra  credit 
for  experience  is  not  a new  one  and  in  fact  has  been  used  for  a number  of 
years  by  the  Federal  Aviation  Administration  (FAA).  While  the  methods  and 
standards  for  evaluating  experience  have  varied  over  the  years,  a common 
element  of  the  experience  scales  used  has  been  their  emphasis  on  aviation- 
related  experience, 

A detailed  account  of  past  and  current  methods  of  assigning  credit  for 
experience  is  available  elsewhere  (1).  However,  a brief  description  of  the 
current  method  used  by  the  CSC  follows.  Fifteen  points  extra  credit  is  given 
to  those  applying  for  FAA  ATC  jobs  who  have  had  at  least  1 year  of  previous 
(usually  military)  radar  (IFR)  ATC  experience.  Ten  points  extra  credit  is 
given  to  those  with  nonradar  (VFR)  ATC  experience  or  less  than  1 year  of  IFR 
ATC  experience,  ground  control  operator  (GCA),  Flight  Service  Station  (FSS), 
or  International  Flight  Service  Station  (IFSS)  experience.  Five  points  credit 
is  given  to  those  who  have  pilot,  air  carrier  dispatcher.  Air  Defense  Command, 
or  navigation/bombardier  experience,  or  have  completed  1 year  of  post-college 
graduate  study,  or  meet  the  Superior  Academic  Achievement  criterion.  The 
ratings  involved  in  determining  how  many,  if  any,  extra  points  are  to  be 
allocated  to  an  individual  based  on  experience  are  very  time  consuming  for  the 
rater,  and  the  rating  itself  is  not  sensitive  to  the  individual's  quality  of 
experience. 

An  FAA  team  headed  by  Dr.  3ohn  T.  Dailey  has  developed  occupational 
knowledge  tests  that  are  designed  to  measure  the  applicability  of  applicants' 
past  experience  to  the  requirements  of  the  FAA  air  traffic  controller 
specialty  (2).  Tests  were  constructed  in  the  hopes  of  replacing  the  current 
rating  system  with  a better  measure  of  the  ATC- related  knowledge  the 
individual  brings  on  board.  If  successful,  an  occupational  knowledge  test 
would  be  a much  more  cost  effective  means  of  accurately  assigning  extra  credit 
than  current  time-consuming  rating  methods.  The  purpose  of  the  present  study 
is  to  determine  whether  or  not  it  would  be  productive  to  use  ATC  Occupational 
Knowledge  Test  (OKT)  101-B  to  assign  extra  credit  in  place  of  the  present  ATC 
Rating  Guide.  In  addition,  the  study  will  make  recommendations  about 
implementation  of  any  new  system  of  assigning  extra  credit  using  an  OKT. 


Validation 


The  sample  used  in  this  study  was  784  ATC  trainees  who  entered  the  FAA 
Academy's  16-week  ATC  training  program  between  July  13  and  December  14,  1976. 
All  trainees  completed  a preemployment  questionnaire,  took  ATC  Occupational 
Knowledge  Test  (OKT)  101-B,  and  remained  in  the  training  program  until  they 
either  passed  or  failed  training.  Of  the  trainees,  701  passed  training  and 
83  failed. 

Based  on  responses  to  a preemployment  questionnaire,  the  trainees  were 
divided  into  three  experience  groups: 

Group  1:  No  prior  creditable  aviation- related  experience. 

Group  2:  Some  prior  aviation- related  experience  (Air  Defense  Command, 
Control  Tower  Operator  rating,  ATC  rating.  Communications  Operator,  Airman 
Certificate) . 

Croup  3:  Prior  VFR  or  IFR,  Center  or  Tower  or  FSS  experience. 

hach  individual  was  also  scored  on  the  OKT  and  was  designated  as  either 
having  passed  or  failed  training.  Of  the  784  Individuals,  250  (32  percent) 
were  classified  in  experience  Group  1,  207  (26  percent)  were  in  Croup  2,  and 
327  (42  percent)  were  in  Group  3.  The  Pearson  correlation  between  OKT  scores 
and  experience  group  was  .64,  and  as  Figure  1 demonstrates,  OKT  scores  clearly 
differentiated  the  three  groups.  The  average  score  (mean)  for  the  three 
experience  groups  on  OKT  were  as  follows:  No  Experience,  52.5;  Some 
Experience,  69.4;  and  ATC  Experience,  77.6. 


Figure  1.  OKT  scores  by  experience  group  1976 
validation  sample. 
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Further,  while  pass/fail  status  correlated  .25  with  OKI,  it  correlated 
only  .12  with  ATC  experience,  and  as  shown  in  Figure  2,  OKI  scores  also 
differentiated  between  passing  and  falling  students:  Passing  Students,  68.8; 
and  Failing  Students,  55.7.  Based  on  this  evidence  it  appears  that  the  OKI 
could  be  used  in  place  of  experience  now  given  extra  credit  on  the  ATC  Rating 
Guide  to  air  traffic  control  applicants.  The  next  step  is  to  determine 
appropriate  cutting  scores  on  OKT  for  assigning  extra  credit. 


Figure  2.  OKT  scores  by  pass/fail  status 
1976  validation  sample* 


OKT  cut  scores.  Table  1 gives  the  OKT  score  distributions  for  the  total 
1976  group,  for  passes  and  for  failures,  for  each  of  the  three  experience 
groups,  and  for  all  groups  combined.  As  can  be  seen  from  the  table,  the 
proportion  of  failures  drops  for  all  experience  groups  when  those  who  scored 
less  than  60  are  compared  with  those  who  scored  60  or  higher  on  the  OKT.  For 
all  experience  groups  combined,  the  next  dramatic  drop  in  failure  rate  occurs 
for  those  whose  OKT  scores  are  greater  than  or  equal  to  75;  if,  however, 
failure  rates  across  experience  groups  are  examined,  other  possible  cutting 
scores  appear.  For  the  No  Experience  group,  the  proportion  of  failures  is 
fairly  constant  until  the  OKT  scores  are  80  or  higher,  but  only  5 percent  of 
the  group  had  scores  in  this  range.  The  failure  rate  for  the  Some  Experience 
group  indicates  that  a cutting  score  of  75  or  higher  might  work  best  for  that 
group,  while  the  data  for  the  ATC  Experience  group  indicate  clearly  that  70  or 
higher  would  be  a good  cutting  score.  Preliminary  scanning  of  the  data 
identifies  three  possible  cutting  scores  for  assigning  credit  based  on  OKT 
scores,  either  70,  75,  or  80  points  or  higher. 

Table  2 shows  the  passes  and  falls  across  experience  groups  when  no  cutting 
score  is  used  and  when  each  of  the  three  cutting  scores  is  used,  for  those  who 
are  above  and  those  who  are  below  the  cut  scores.  As  can  be  seen  from  Table 
2,  53.7  percent  of  the  sample  scored  above  70,  42  percent  scored  75  or  above. 
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Total  Pass  Fail  % Fall  Total  Pass  Fall  X Fail 

All  Groups  204  199  5 2.5  All  Groups  580  502  78  13.4 

No  Experience  12  12  0 0.0  No  Experience  238  196  42  17.6 

Some  Experience  42  41  1 2.4  Some  Experience  165  150  15  9.1 

AXC  Experience  150  146  4 2.7  AlC  Experience  177  156  21  11.9 


and  26  percent  scored  above  80.  The  proportion  of  failures  was  reduced  from 
10.6  percent  in  the  overall  sample  to  5.5  percent,  3.0  percent,  and  2.5 
percent,  respectively,  for  the  three  cutting  scores.  It  is  evident  that  the 
.5  reduction  in  percentage  of  failures  gained  by  going  from  a cut  score  of  75 
to  a cut  score  of  80  does  not  justify  the  16- percent  drop  in  the  proportion  of 
trainees  above  the  latter  cut  score.  The  decision  on  appropriate  cutting 
scores  is  narrowed  down  to  being  between  70  and  75. 

The  evidence  indicates  that  a cutting  score  of  75  is  consistently  better 
than  the  cut  score  of  70,  although  the  differences  are  not  dramatic.  It  is 
possible  that  the  differences  observed  between  the  two  cutting  systems  will 
not  hold  up  under  cross-validation.  In  order  to  determine  this,  a second 
sample  of  432  trainees  who  entered  and  completed  training  in  1977  and  had  taken 
the  OKT  was  evaluated. 


Cross- Validation 

There  were  432  trainees  in  the  cross-validation  sample  who  entered 
training  between  January  11,  1977,  and  March  8,  1977,  and  who  either  passed  or 
failed  training,  had  taken  the  OKT,  and  for  whom  experience  information  was 
known.  Table  3 shows  the  number  of  those  who  passed  and  failed  and  the 

TABLE  3 


Cross-Validation  Sample  1977 


Group 

Total  N 

Pass 

Fail 

% Fail 

Total 

437 

376 

56 

13.0 

No  Experience 

124 

99 

25 

20.0 

Some  and  ATC  Experience 

308 

277 

31 

11.0 

Greater  than  or 
equal  to  ,70  on  OKT 

253 

240 

13 

5.1 

Less  than  70  on  OKT 

179 

136 

43 

24.0 

Greater  than  or 
equal  to  75  on  OKT 

185 

178 

7 

3.8 

Less  than  75  on  OKT 

247 

198 

49 

19.8 

proportion  of  failures  for  the  following  groups:  (1)  the  total  group,  (2) 
those  with  experience,  (3)  those  without  experience,  (4)  those  who  scored  at  or 
above  the  OKT  cut  scores  of  70  and  75,  and  (5)  those  who  scored  below  the  cut 
scores  of  70  and  75.  As  can  be  seen  from  Table  3,  the  data  from  the  1977 
sample  is  consistent  with  that  from  the  1976  sample.  The  proportion  of 
failures  for  those  who  received  credits  for  experience  is  11  percent  while  the 
failure  rate  drops  to  5.1  percent  for  those  who  scored  above  70  on  OKT  and  to 
3.8  percent  for  those  who  scored  above  75.  As  shown  in  Figure  3,  OKT  again 
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Figure  3.  OKI  scores  by  experience  group 
1977  cross-validation  sample. 

clearly  differentiated  experienced  individuals  from  those  with  no  experience. 
The  correlation  of  experience  and  OKT  was  .67.  Figure  shows  the  proportion 
of  passing  and  of  falling  students  who  scored  at  the  same  level  or  above  OKT 
scores  for  the  1977  sample.  Again,  the  OKT  scores  separate  the  groups  and 


Figure  4.  OKT  scores  by  pass/fail  status 
1977  cross-validation  sample, 


correlate  with  pass/fail  status  .28.  Clearly,  the  use  of  the  Occupational 
Knowledge  Test  would  be  an  Improvement  over  the  use  of  experience  (which 
correlated  with  pass/fail  status  .18)  as  a means  of  assessing  an  applicant's 
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ATC- related  knowledge  prior  to  training.  OKI  scores  appear  to  be  very  closely 
related  to  prior  experience,  yet  at  the  same  time  are  more  sensitive  to  those 
aspects  of  prior  experience  tf.at  relate  to  success  in  ATC  training. 

OKT  Cut  Score 

The  ultimate  decision  on  which  cut  score  to  use  on  the  OKT  is  a subjective 
one.  If  the  data  for  1976  and  1977  are  combined,  a cut  score  of  70  would 
result  in  a failure  rate  of  5.0  percent  for  those  getting  credit,  while  a 75 
cut  score  would  result  in  a 3.3-percent  failure  rate.  If  we  assume  that  there 
will  be  1,600  trainees  during  the  year,  and  that  the  proportions  failing  and 
scoring  at  or  above  70  and  75  on  OKT  will  remain  relatively  constant,  we  can 
project  the  effect  of  this  cutting  system  over  a year.  If  a cutting  score  of 
70  is  used,  887  of  the  trainees  would  get  extra  points,  and  45  of  those 
trainees  would  fail.  If  a cutting  score  of  75  is  used,  676  trainees  would  get 
points  and  22  of  those  trainees  would  fail.  This  means  that  211  people  who 
would  pass  and  get  extra  points  with  a 70  cutoff  score  would  not  get  extra 
points  with  a 75  cut  score.  Twenty- three  of  the  people  who  would  get  extra 

points  with  a 70  cutoff  but  would  fail,  would  not  get  extra  points  with  the  75 

cut  score.  Approximately  11  percent  of  the  individuals  who  scored  between  70 
and  75  on  the  OKT  would  subsequently  fail  training,  indicating  that  a meaningful 
savings  could  be  made  by  using  a cut  point  of  75  instead  of  70  on  the  OKT.  If 

prior  experience  were  used  to  assign  extra  credit,  1,108  of  the  1,600  students 

would  receive  extra  credit  and  95  of  them  would  fail.  Based  on  the  data  cited 
above,  it  is  recommended  that  the  OKT  with  a cut  score  of  75  be  used  to  assign 
extra  credit  for  the  selection  of  air  traffic  controllers,  in  place  of  the 
current  experience  scale. 

Implications 

The  results  of  this  study  indicate  that  the  OKT  could  be  a practical  means 
of  assigning  extra  credit  to  ATC  applicants  with  useful  aviation  knowledge.  It 
appears  that  the  OKT  would  improve  current  methods  for  allowing  extra  credit 
for  experience  because  it  relates  to  the  quality  rather  than  just  the  quantity 
of  past  experience.  Because  the  OKT  is  very  highly  related  to  prior  experience, 
many  of  the  people  receiving  credit  for  past  experience  under  the  current 
procedure  would  also  receive  extra  credit  based  on  the  OKT.  However,  many  of 
those  individuals  currently  receiving  extra  credit  for  experience  who 
subsequently  fail  training  would  not  receive  extra  credit  if  OKT  scores  were 
used. 

The  use  of  the  OKT  to  replace  the  assessment  of  previous  experience  would  be  a 
very  cost  effective  means  of  assigning  extra  credit.  It  would  minimize  the 
need  to  review  CSC  Form  SF-171's  to  determine  the  applicants'  relevant 
experience  and  would  reduce  the  number  of  work  hours  required  to  assign  extra 
points.  In  addition,  the  proportion  of  individuals  who  now  receive  extra 
credit  based  on  the  ATC  Rating  Guide  and  who  later  fail  training  would  be 
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reduced  from  7.6  percent  to  3.8  percent.  Thus,  it  is  recommended  that  the  OKT 
with  a cut  score  of  75  be  used  in  place  of  the  experience  scale  to  assign 
extra  credit. 


Additional  Research 

The  Federal  Aviation  Administration  in  conjunction  with  the  Civil  Service 
Commission  is  in  the  process  of  evaluating  changes  to  the  CSC  ATC  selection 
test  battery.  As  part  of  this  effort,  the  OKT  will  be  examined  to  determine  if 
it  makes  a meaningful  contribution  to  a revised  battery,  and  if  so,  how  many 
additional  points  for  ATC  job- related  experience  should  be  allowed  based  on 
OKT  scores.  It  is  probable  that  the  OKT  will  continue  to  make  a meaningful 
contribution  to  any  selection  procedure  since  the  correlations  of  OKT  with 
other  possible  battery  components  are  very  small.  It  appears  that  the 
Occupational  Knowledge  Test  is  a valid  measure  of  those  experiences  that  an 
individual  brings  on  board  that  are  related  to  success  as  an  FAA  air  traffic 
controller. 
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